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INTRODUCTION 

The School District of the City of Saginaw first used the "Success in 
Beginning Reading and Writing" (SBRW) materials during the 1986-87 school year 
in fifteen Junior First classrooms. Based on that experience, these materials 
and their associated techniques were piloted in four first grade classrooms at 
three different buildings. This report is based upon the first grade pilot 
and comparison classrooms at the same three buildings that used basal readers 
and workbooks. Up to this point, the basal reader and its workbook were the 
district-wide adopted approach for the instruction of reading and writing. 
This pilot, if successful, could lead to a major change in the district's 
reading and writing curriculum. 

Because many of the activities found in workbooks may be of dubious value 
in teaching reading, the SBEW program allocates 90 percent of the child's 
reading time to actually reading and writing. The SBKW program is one of the 
few eclectic approaches to teaching reading. The program itself stems from 
the belief of Duke University researchers that children should be taught to 
read and write using materials they will rely on later in life. Reading and 
writing complement each other in the program and children have the chance to 
do both during a daily two hour period. The SBRW program depends heavily on 
the children's'use of language and the building of a good experiential back- 
ground before any reading or writing instruction ever begins. 

The SBRW program developers claim their program is rewarding for stu- 
dents, teachers, and administrators. By the end of the program, pupils have 
been exposed to a wide variety of printed material and have composed many 
types of written communication. Most important, these young learners have 



been given the opportunity to feel good about themselves because they have 
been successful • 



Teachers as wsll find the SBRW a strong program according to the program 
proponents. It takes advantage of their expertise and ingenuity by asking 
them to develop a brief outline into a challenging and rewarding experience 
for the pupils. 

Again according to program developers, administrators can see the SBRW as 
cost effective. No ditto masters, workbooks, basal readers, or vocabulary 
charts are required since materials used in the program are readily available. 
Subscriptions to several magazines and the daily newspaper and a well-stocked 
library constitute the supplies needed. Thus monies once spent on kits and 
materials can be used more effectively in other areas. 

Overall, the SBEW program according to program literature has eight dis- 
tinctive characteristics. They include the following: 

1. There is no predetermined sequence of skills, although 
skills are emphasized in all modules. The timing for 
teaching certain skills is often generated within the 
moment — to extend pupils'' social, psychological, and 
mental perspective at the optimal point in the learn- 
ing process. 

2. Sight words are not taught from isolated lists, but as 
they appear in a sentence or paragraph and in a mean- 
ingful context. 

3. Verbal communication plays an important role in chil- 
dren^'s understanding. Word meanings are taught as 
they are volunteered by students in their own phrases 
or sentences. 

4. Students' vocabularv is displayed on a chart, a key 
element of the SHEW program and an identifiable 
feature of these classrooms. 

5. Students begin with words they already know and pro- 
ceed to learn words volunteered by others in the class- 
room or found somewhere in print. This freedom to 
learn to read and write an unlimited and uncontrolled 
vocabulary is another feature of the program. 



6. Students get off to a successf ill , start because they 
are not afraid of failure* 

7. Small groups are formed from time to time, but never 
on the basis of abf.lity levels, and are maintained 
only until predetermined objectives are realized* 

8. Students' positive self-concepts develop from success- 
ful endeavors in reading and writing. 

The evaluation of SBRW pilot at the first grade level will involve two 
parts. Part 1 deals with self report data from pupils, teachers, and 
principals concerning the SBRW pilot contrasted with the workbook approach. 
Part 2, the subject of this report, will deal with a pre- to post-test 
[ California Achievement Tests (CAT) — Form E] comparison of the SBRW program to 
the Holt Basic Reading (HBR) program and the Spring, 1985 national norming 
group of the CAT. 

What follows are the specifics of the Part 2 evaluation Into the effec- 
tiveness of the SBRW treatment as contrasted with the HBR prograa and the CAT 
national noming group. 
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EVALIMlTION PROfJEDORES 



In conducting an evaluation of an educational program, information is 
sought concerning the process and the quality of outcomes of that educational 
program. The processes and outcomes can be compared either to predetermined 
standards or competing educational programs. 

This evaluation of SBRW uses the Spring;, 1985 CAT national norraing group 
at kindergarten and first grade as the standard. Initially it was planned to 
use the Holt Basic Reading Program (HBR) as the competing educational program 
or standard. An analysis of kindergarten test scores in vocabulary, reading 
comprehension, reading total, and language expression on the California 
Achievement Tests (CAT) Form E revealed no statistically significant differ- 
ences (alpha =« .05) in the test scores existed between SBRW (pilot) and HBR 
(control) pupils at the onset (see Appendix A for further details of the 
analysis). A gap reduction evaluation design was chosen and upon further 
analysis of the results it was found that the HBR group was inappropriate as a 
control because of its poor performance. This poor performance was graphed 
using the gap reduction evaluation design (see Appendix A for these figures). 
The poor performance of the HBR control is evident from the graphs. The CAT 
national norming group (NNG) at the kindergarten and first grade levels shows 
a continuing upward growth that would be expected of a rea.\isr;ic control or 
comparison group. 

The pre- to post-test results on the CAT in vocabulary, reading compre- 
hension, reading total, and language expression served as the educational 
outcomes of interest. Treatment students were pre-tested in April/May, 1988. 
Pupils lacking a post-test score (six treatment pupils) were given a post-test 
score by adding the average post-tesc gain to their initial pre-test score as 
recommended by Tallraadge, G. , et al. (1987). 
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The predetermined standard chosen to compare the two groups was the 
Relative Growth Index (RGI)# This index indicates the percentage increase or 
decrease of the treatment group (SBRW) as compared to the comparison group 
(NNG) between the mean pre- and post-test achievement levels. Thus the index 
allows us to make inferences about program effectiveness that are limited to 
relative rather than absolute impacts. It is expected that the gap xiill stay 
the same or be reduced as a result of the SBRW program participation. Figure 
1 below illustrates the reduced gap expected between treatment (T) and com- 
parison (C) groups. 

Prc-Test Post-Test 




T C T C 

Figure 1. Gap ledactloa Design. 
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To calculate the Relative Growth Index (RGI), the comparison group's pre- 
and post-test standard deviations are pooled* This pooled standard deviation 
is the metric in wliich growth estimates for the pilot and ccroparison groups 
are cast. Finally, the growth of the pilot group is expressed c5 a percentage 
of the growth of the comparison group, thus providing an easy-to-interpret 
Relative Growth Index (RGI). (See Appendix B for the steps involved in the 
calculation of this index.) RGI's less than 100% indicate that the SBEW 
pupils fell further behind the NNG during this school year. RGI's equal to 
100% indicate that the pilot group grew at the same rate as NNG comparison 
students, and RGI's greater than 100% indicate that SBRW participants out 
gained the NNG participants. A negative RGI means that one group grew while 
the other group lost. 

The study's alternate hypotheses are stated below. 

1. There will be a Relative Growth Index (RGI) of 100% 
or greater in reading vocabulary as measured by CAT 
for SBRtf participants. 

2. There will be a Relative Growth Index (RGI) of 100% 
or greate.T in reading comprehension ^s measured by 
CAT for SliRM participants. 

3. There will be a Relative Growth Index (RGI) of 100% 
or greater in reading total as mee red by CAT for 
SBRW participants. 

4. There will be a Relative Growth Index (RGI) of 100% 
or greater in language expression as measured by CAT 
for SBRW participants. 

The null hypothesis in each case above is that there will be a Relative 
Growth Index (RGI) less than 100% in each subtest of the CAT for SBRtf partici- 
pants rcgarr^'^ss of sign. 



PRESENTATION OF DATA 

The criterion chosen to determine the success of the SBRW program was the 
Relative Growth Index (RGI). The RGI is the statistic used in the gap reduc- 
tion evaluation model design. The research question posed is "Whether the 
SBRW pilot group is keeping up with or surpassing the NNG comparison group." 
Tha gap measured is the gap between the mean achievement level of the SBRW 
group and the mean achievement level of the NNG comparison group. It is 
hypothesized that the gap between the pilot and comparison grpups will remain 
the same or be reduced between pre- and post-testing. A total of 59 SBRW and 
approximately 20,000 NNG pupils results entered into the calculation of RGI 
statistics (see Appendix C for the summary statistics used to calculate the 
RGIs). 

To evaluate this hypothesis the NNG comparison group's pre- and post-test 
standard deviations are pooled. This pooled standard deviation is then used 
as the metric in which growth estimates for the pilot and comparison group are 
measured. Finally, the growth of the pilot group is expressed as a percentage 
of the growth of the comparison group, thus providing an easy-to-interpret RGI 
(see Appendix B for the exact steps to calculate the Relative Growth Index). 

The interpretation of the RGI deserves a bit of an explanation. A RGI 
less than 100% indicates the SBRW pilot group is falling behind the comparison 
group. When the RGI equals 100% it signifies the pilot group is keeping equal 
to the NNG comparison group. A RGI greater than 100% means the pilot group is 
catching up to the comparison group. Figure 2 puts this interpretation in 
graphic form relative to the gap between the pilot and comparison group. 
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• RGI less Chan lOOZ 

signifies falling behind 
eoQparison group 



• J^GI equal to lOOZ 
signifies keeping up 
wich comparison group 



• RGI greater than lOOZ 
signifies catching up 
Co comparison group 



e Post 



Figure 2. Interpretatloa of Relative Growth Indices (RGIs). 



Table 1 below presents the RGIs by hypothesis and subtest. 



TABIZ 1. RELATIVE QMWTH INIKX (RGI) BY HYPOTHESIS 
AND SUBTEST AREA. 



Hypo thesis/ Sub test 
Number / Area 


Relative Growth Index 


1. 


Heading Vocabulary 


-3 7. 03% 


2. 


Reading Comprehension 


69.14% 


3, 


Reading Total 


-22.78% 


4. 


Language Expression 


6.29% 




High 



Test 
Score 



Low 



comparison group 
pilot group 



pie pi: 



Pdst 



High 



Test 
Score 



Low 



Pte 



comparison group 
pilot group 



P6st 



High 



Test 
Score 



Low 



comparison group 
pilot group 



A review of Table 1 reveals that the SBRM (pilot group) fell behind the 
NNG (comparison group) on all four subtests. The SB'RW group failed to close 
the gap between the comparison group in any of the four comparisons Appendix C 
presents a graph of each of the comparisons* 

In the areas of reading comprehension and language expression, the SBRW 
group grew (RGI = 69.14% and 6,29% respectively) but not as fast as the com- 
parison group • Thus the gap increased but the SBFW experienced growth* A 
much worse situation was evident in reading total and reading vocabulary with 
the SBRW group losing (RGI = -22* 7 8% and -3 7,03% respectively) as the compari- 
son group continued to experience progress* See Appendix C for a graph by 
subtest area for SBFW pilot versus the comparison NNG using relative growth 
statistics. 

The following chart specifies the hypotheses relating to the RGI's and 
their status relative to the research question. 



As indicated above the SBRW pilot group failed to equal the growth of the 
NNG group in any of the subtest areas. Thus the SBEW program appears to be 
less successful generally than other programs nationally in bringing about 
achievement gains on the CAT on the four subtests reviewed. 



Hypothesis 
# 



Subtest Area 



Results Which 
Equal or Exceed Gains 
Hypothesized 



1 
2 
3 
4 



Reading Vocabulary 
Reading Comprehension 
Reading Total 
Language Expression 



No 
No 
No 
No 
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SUMMARY 

The School District of the City of Saginaw is completing a pilot of a new 
reading and writing program in first grade* The new program entitled, 
"Success in Beginning Reading and Writing'' (SBRW) is based on the philosophy 
that children should be taught to read and write using materials they will 
rely on later in life* SBRW is thus philosophically different than a basal 
reader and workbook as presently employed nationally in most school districts* 

This report has contrasted the SBRW and the CAT national norming group 
(NNG) through the use of the Relative Growth Index (RGI)* This index indi- 
cates the percentage the pilot group (SBRW) is falling behind or catching up 
to the comparison group (NNG) in terms of mean pre- and post-test achievement 
levels* The SBRW program showed itself to be an inferior program in terras of 
fostering growth from pre- to post-testing in all four CAT subtest areas — 
language expression, reading total, reading vocabulary, and reading compre- 
hension — than the NNG comparison group* The increased gap (or the RGI less 
than 100%) for the SBRW pilot group were 69*14%, 6*29%, -22*78%, and -37*03% 
for reading comprehension, language expression, reading total, and reading 
vocabulary, respectively* 

Overall, it was found that the SBRW pilot failed to reduce the academic 
achievement gap between the California Achievement Test (CAT) national norming 
group in the general areas of reading and language expression * 

What follows is a series of recommendations relative to the SBRW program* 
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RECOHMENDATIONS 



Listed below are a series of recommendations based on the findings from 
both this report (Part 2) and its companion report (Part 1) finished earlier. 
These recommendations are offered in an effort to improve the implementation 
and study the impact of SBFW in the future. 

!• From a review of, the process findings presented 
in the Part 1 report and the outcome findings 
presented in the Part 2 report, it appears the 
SBFW pilot might be continued. Before making 
this decision, additional research should be 
done into evaluation findings of the program 
developers to determine whether what looks as 
a questionable pilot should be continued and how 
it might be further enhanced to make it worth 
to be continued. 

2. If the pilot does continue, definite plans for 
any future or continuing study of the SBRW pro- 
gram should be outlined in advance. Such 
advanced planning would yield more comprehen- 
sive accounting of all participants that ham- 
pered this study. 
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APPENDIX A 



TO: Miriam Sweigart 

FROM: Richard Glaus 
Raul Rio 

RE: An Analysis of the Equivalency o£ Pilot and Control Sites: 
Success in Reading and Writing 

DATE: December 21, 1987 



Ue have just completed a preliminary analysis of the data you pro- 
vided regarding the Success in Reading and Writing Project. The 
major findings and recommendations are presented below. 



Hajor Findings ; 

* As indicated on the following page, a total of 208 
pupils are participating in the control and pilot 
classrooms. 

- Of these students, 89 pupils are receiving the 
Project treatment (pilot), and 119 pupils are 
not (control). 

- California Achievement Tests (CAT) data are 
available on a total of 139 children. Of 
these tested, eighty of the 119 (67. 2Z) 
pupils are not receiving the pilot treatment, 
and 59 of the 89 (66.3Z) pupils are receiving 
the treatment. 

* Data by gender and racial/ethnic classifications 
were also tabulated and are presented on the 
following page. 

- For those students on whom CAT data were avail- 
able, no statistically significant differences 
in the test scores exist between those students 
receiving the treatment and those students who 
are not. 
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APPENDIX' A 



Miriaa Sweigact 
Page 2 

December 21, 1987 



Reco—endatloim : 

In order to insure that differences can be detected at the conclu- 
sion of this Project, the following recommendations are offered: 



- Project activities as well as objectives and 
expected outcomes must be described in detail. 
These Project activities must be standardized 
across buildings, teachers and groups. 

- Monitoring of Project activities should be ^ 
conducted. 

- Every effort must be made to test both pilot 
and control pupils in the Spring of 1988. 



Ue will be looking forward to assessing the outcome of this 
Project. We will also be willing to assist you on a time avail- 
able basis. If you have any questions, please contact us. 



cc: William Cheaney 
Barry E. Quimper 
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APPENDIX -A 

TOTAL BIDMBER OP STODEMTS BY PARTICIPATIOH STATUS AHD BDIL0IH6 

Participation Status Building Count 

Control Emerson 32 

Control Heavenrlch 23 

Control He rig 44 

Subtotal • . • 119 

Pilot Emerson 24 

Pilot Heavenrlch 45 

Pilot Herlg 20 

Subtotal 89 

TOTAL 208 



HimBER OP STDDBflS WITH TEST SCORES BY PABTICIPATIOH STATUS AHD BglLDDC 

Participation Statu s Building Count 

Control Emerson 31 

Control Heavenrlch 9 

Control Herlg 40 

Subtotal 80 

Pilot Emerson 13 

Pilot Heavenrlch 29 

Pilot Herlg 17 

Subtotal 59 

TOTAL 139 
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APPENDIX' A 

J 

ndmbu of sOTDBrfs yriH test scores by gkhder 

Gender Participation Status Count 

Fema le 8 Con t r ol 3 8 

Females PlloC 25 

Subtotal 63 

Males control 42 

Males Pilot 34 

Subtotal 76 

TOTAL 139 

NUMBER OF STODniTS WITH TEST SCORES BY EIHHIC (aOOP 

Racial/Ethnic Group Participation Status Count 

Caucaslon Control 38 

Caucasloa Pilot 13 

Subtotal ..... 51 

Hxspanlc Control 3 

Hispanic Pilot 9 

Subtotal . . . . , 12 

Black Control 39 

Black Pilot 37 

Subtotal 7^ 

TOTAL 139 
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APPENDIX A 



wiantv A 1 REIATIVB GEOHTH OF THE SUCCESS IM BEGIHHIIIG READING AND WRITING (SBBW) PIUW GROUP 

^' ialS^J^Hotf BASIC Zding (hbe) control group from pre- to post-testing 

IN READING VOCABULARY. 
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APPENDIX A 

FIGUBK A 2. REIATIVB GROWTH OF THE SOCCBSS IM BEGIHIiING BKAOING AND URITING (SBB#) PILOT GROUP 
* VERSUS THE HOLT BASIC READIMG (BBR) COMIROL GROUP PROM PRE- TO PC^T-TESTING 

IN REAOIMG COHFREHBHSIOM. 
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FIGOEE A. 3. SEIATIVE GEOHTH Hi TBE SDCCESS IN BEGINNING READING AND IfRITING (SBW) PILOT GKOUP 
TOBSI7S THE HOLT Ri^IC BEADING (HBB) CONIBOL GBOUP FBOK FBB- TO POST-TESTING 
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FIGOBS A.4. BSLAIIVE GROyTH OP THK SUCCESS IM BEGUUIliG READING AND HKITING (SBBI) PILOT GROUP 
VERSUS THE HOW BASIC READING (HBR) CONTROL GROUP FROM PRE- TO lOST-TESiING 

IN fAH'Tfffr'^ EIFRESSION. 



to 

-O" 



ERIC 



p 

E 
R 
F 
0 
R 
M 
A 
.N 
C 
E 

L 
E 
V 
E 
L 



29 



.6-- 

.5 

.4 4- 

.3 

.2 
.1 4- 

.1 4- 

.2 
,3 

.4 4- 
.5 
.6 4> 



Pre-Test 
Gap = 
0.170 




Post 
Gap = 
-0. 



\ Control Group's 
( Standardized 
Growth 
Estimate 
-0.250 



-Test I 
130 ^ 



0 



> 

w 
a 

M 

X 

> 



Pilot 
Group ' s 
Stan- 
dardized 
Growth 
Estimate = 
0.050 



+ 



PRE-TEST 



POST-TEST 



TESTING TIME 



Le gend 



Pilot Group 



Control Group 



30 



APPENDIX B 



STEPS TO CALCDLATE THE RELATIVE GROWTH INDEX IN THE GAP 
REDUCTION RESEARCH DESIGN 



STEP 1: (For use with a standardized achievement test*) Convert each 



project and comparison group student's raw pre-test and post-test 
scores to scale scores using the correct conversion table for the 
form and level of the test used* If a non-standardized test was 
used skip this step. 



STEP 2: Compute the mean pre-test and post-test (raw or, if the test has 
them, scale) scores of project students at each grade level. 

STEP 3: Compute the mean pre-test and post-test scores of comparison group 
students at each grade level. 

STEP 4: Compute the pre-test and post-test standard deviations of comparison 
group students at each grade level. 



Subtract the project group's mean pre-test score from the comparison 
group's mean pre-test score. Divide the difference by the compari- 
son group's pre-test standard deviation and label the result the 
pre-test gap. 

Subtract the project group's mean post-test score from the compari- 
son group's mean post-test score. Divide the difference by the com- 
parison group's post-test standard deviation and label the result 
the post-test gap. 



STEP 7: Subtract the post-test gap (from Step 6) from the pre-test gap (from 
Step 5) and label the difference the gap reduction. (The gap reduc- 
tion may be negative. Be sure to keep track of the sign!) 



Subtract the comparison group's mean pre-test score from its mean 
post-test score and label the difference the comparison group's 
unstandardized growth estimate. 

Using the comparison group's pre- and post-test standard deviations, 
calculate the following value: 



Label this value the comparison group's pooled standard deviation. 




2 



21 3i 



APPENDIX B 



STEP 10: Divide the comparison group's unstandardized growth estimate (from 
Step 8) by the comparison group's pooled standard deviation (from 
Step 9). Label this value the comparison group's standardized 
growth estimate. 

STEP 11: Add the gap reduction (from Step 7) to the comparison group's stan- 
dardized growth estimate (from Step 10). Label this sum the project 
group's standardized growth estimate. 

STEP 12: Divide the project group's standardized growth estimate (from Step 
11) by the comparison group's standardized growth estimate (from 
Step 10). Multiply the result by 100 to convert it to a percent and 
label it the Relative Growth Index (RGI). 
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CALCULATION OF THE RELATIVE GROWTH INDEX (RGI) IN THE GAP 
REDUCTION RESEARCH DESIGN 

An example may help operationalize the steps on the preceding two pages. Con~ 
slder the following data set for a particular grade level. 



Pretest Mean 
Pretest Std. Dev. 
Posttest ^tean 
Posttest Std. Dev. 



Project 
Group 

355.34 

N/A 
365.88 

N/A 



Comparison 
Group 

361.63 
10.48 
370.63 
9.50 



STEP 5: 
STEP 6: 
STEP 7: 
STEP 8: 

STEP 9: 



(361.63 - 355.3/,) t 10.48 - 6.29 f 10.48 = .60 = the pretest gap. 
(370.63 - 365.88) ^ 9.50 = 4.75 9.50 - .50 = the posttest gap. 
.60 - .50 = .10 = the gap redaction. 

370.63 - 361.63 = 9.00 = the comparison group's uustandardized growth 
estimate. 



'(10.48)^ + (9.50)^ 



109.83 + 90.25 



STEP 10: 
STEP 11: 
STEP 12: 



^100.04 = 10.00 = the comparison group's pooled standard 

deviation. 

9.00 7 10.00 = .90 = the comparison group's standard growth estimate. 
.90 + .10 = 1.00 = the project group's standardized growth estimate. 
(1.00 .90)100 = lllZ a the Relative Growth Index (RGI). 
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TABLE C.l. SUMMARY STATISTICS USED TO CALCULATE THE RELATIVE 
GRWTH INDICES (RGIs) BY SUBTEST AND (SIOUP. 









ouuiuia ry 


Ota tistics 




Subtest 


Group 


Pre- 


-Test 


Post- 


-Test 




iManie li 


Me an 


S.D. 


Mean 


S.D. 


Reading 
Vocabulary 


Pilot 59 
Control ~ 20,000 


502.25 
521.00 


N. A.* 
7 9.00 


472.38 
529.00 


N.A. 

79.50 


Re ading 
Compre hension 


Pilot 59 
Control 2i 20,000 


462.32 
501.00 


N. A. 

69.70 


472.11 
539.00 


N.A. 

96.30 


Reading 
Total 


Pilot 59 
Control ^ 20,000 


482.37 
514.00 


N.A. 

67.70 


472 . 45 
538.00 


N.A. 

74.30 


Language 
Expression 


Pilot 59 
Control/v^ 20,000 


509.06 
545.00 


N.A. 

83.80 


510.27 
569.00 


. N.A. 
84.20 



*N.aV''^== Not Applicable • 
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FIGOBE C.l. KEIATIVE GROHTH OF THE SUCCESS IN BBGimiUiG EEAOIHG AND tfRITING (SBSW) PILOT GROUP 
VERSUS THE HOLT BASIC REAOING (HBR) CONIBOL GROUP FROM PRE- TO POST-TESTING 

m REAOING VOCABULARY. 
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PIGDEE C.2. RELATIVE GROHTH OF THE SUCCESS Dl SEGUUUIG REAODiG AND WRITING (SBRH) PILOT GROUP 
VERSUS THE HOLI BASIC READING (HBR) CONTROL GROUP FROH PRE- TO POST-TESTING 

IN READING COMFBEBEHSION. 
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PIGUBE C.3. lEUTIVK GKOUTH OP TUB SUCCESS IN BEGUWING READING AND WRITING (SBW) PILOT GROUP 
VERSUS THE HOLT BASIC READING (HBR) COmOL GROUP PROH PRE- TO POST-TESTING 

IN READING TOTAL. 
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PIGDBS C.4. lEIATIVB GROnTH W THE SUCCESS IM BBGUiHIiiG READING AND WRITING (SBRU) PILOT GROUP 
VERSUS THE BOIX BASIC READING (HBR) CONTROL GROUP FROM PRE- TO POST-TESTING 

IN LANGUAGE EXPRESSION. 
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